<html>
<head><meta charset="utf-8"><title>Async Derivation and Serializable Futures · wg-async-foundations · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/index.html">wg-async-foundations</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/Async.20Derivation.20and.20Serializable.20Futures.html">Async Derivation and Serializable Futures</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="245645498"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/Async%20Derivation%20and%20Serializable%20Futures/near/245645498" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoey <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/Async.20Derivation.20and.20Serializable.20Futures.html#245645498">(Jul 12 2021 at 04:59)</a>:</h4>
<p>A few years ago, I had been experimenting with cooperative, resumable multi-tasking on embedded systems, using hand-written coroutine state-machines called <code>TaskHandle</code>s.</p>
<p>When I learned that the<code>Future</code>s and <code>async/await</code> ecosystem had reached a usable state, I was curious if I'd be able to implement my <code>TaskHandle</code> system in terms of <code>async</code> such that they could be composed with <code>await</code> and Futures primitives rather than the clunky AST-builder-like chaining syntax and hand-written state-machine enums I'd been using. </p>
<hr>
<h2>Resumption</h2>
<p>The problem I ran into is that he system in question - a lightweight chip on battery power - must go down for halt and resume from the prior state when it wakes. The amount of non-volatile storage available is significantly less than its operating memory, and is extremely costly to write, in relative terms of power. This prevents a simple solution such as "just save all memory to disk as a snapshot".</p>
<p>My pre-async system solved this by use of <code>serde</code>, serializing the task tree before a shutdown and deserializing on wake.</p>
<h2>Experimentation</h2>
<p>I had hoped to use <code>async fn</code> syntax to implement these futures, but there appears to be no way to access the generated state machine types to implement <code>Serialize</code>/<code>Deserialize</code> upon them.</p>
<p>I then tried to make a supertrait of Future called <code>SerializeFuture</code>, which required that anything crossing an <code>await</code> barrier was also <code>SerializeFuture</code>. This seemed viable, but the only way I was  able to implement <code>SerializeFuture</code> instances was by hand, then using <code>#[derive(Serialize)]</code>.</p>
<p>With hand-written futures, it was possible to call serialize on a future, then deserialize it again later, but this required full knowledge of that future tree's type, which made <code>BoxFuture</code> intractable for libraries without polymorphic deserialization hacks.</p>
<hr>
<h2>Solution ideas</h2>
<ul>
<li>
<p>A) Somehow invoke a procedural macro on the state machines produced by <code>async fn</code> syntax; possibly just <code>#[annotation]</code> before the <code>async</code>? </p>
</li>
<li>
<p>B) Specify, at the <code>async fn ...</code> declaration site, that additional bounds apply to futures awaited within its state machine, and require that such bounds have a derivation available to derive them on the produced generator, e.g. <code>async(Serialize, Deserialize) fn ...</code></p>
</li>
<li>
<p>C) Allow the <code>impl Future</code> site to attempt to extend its bounds to match the receiver if they are not met using available derivations that map to such bounds</p>
</li>
</ul>
<p>Wakers are hard to serialize, but - unless I'm mistaken - the easiest answer is to simply not bother; calling poll on the root future after deserialization is sufficient to re-schedule all wakers for futures that could be serialized.</p>
<p>This feature may allow us to request a <code>Debug</code> instance for a future, and automatically generate meaningful descriptions of the state machine of an async coroutine, allowing debugger tooling a better means of introspection into asynchrony.</p>



<a name="245647829"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/Async%20Derivation%20and%20Serializable%20Futures/near/245647829" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Kestrer <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/Async.20Derivation.20and.20Serializable.20Futures.html#245647829">(Jul 12 2021 at 05:54)</a>:</h4>
<p>Part A would be difficult because async blocks are not always convertable to Rust code. For example:</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="k">async</span><span class="w"> </span><span class="k">fn</span> <span class="nf">future</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="p">(</span><span class="n">val</span>: <span class="nc">T</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="n">val</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="n">tokio</span>::<span class="n">task</span>::<span class="n">yield_now</span><span class="p">().</span><span class="k">await</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>This is not possible to express as a state machine.</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="k">enum</span> <span class="nc">Future</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="n">Initial</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">val</span>: <span class="nc">T</span><span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w">    </span><span class="n">First</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="n">val</span>: <span class="nc">T</span><span class="p">,</span><span class="w"></span>
<span class="w">        </span><span class="n">r</span>: <span class="kp">&amp;</span><span class="cm">/* what lifetime here? */</span><span class="w"> </span><span class="n">T</span><span class="p">,</span><span class="w"></span>
<span class="w">        </span><span class="n">awaited</span>: <span class="nc">impl</span><span class="w"> </span><span class="n">Future</span><span class="o">&lt;</span><span class="n">Output</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">()</span><span class="o">&gt;</span><span class="p">,</span><span class="w"></span>
<span class="w">    </span><span class="p">},</span><span class="w"></span>
<span class="w">    </span><span class="n">Final</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>



<a name="245648913"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/Async%20Derivation%20and%20Serializable%20Futures/near/245648913" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoey <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/Async.20Derivation.20and.20Serializable.20Futures.html#245648913">(Jul 12 2021 at 06:15)</a>:</h4>
<p>Would it be possible to pick a very naive approach with room for later nuance, by stating that <code>first</code> as a future itself must implement the bound requested, in order to be eligible to receive it for the latter? As in - only the outer layer gets exposed to the proc-macro, and leaves stubs for the child futures?</p>
<p>Additionally, isn't <code>r</code> just an alias of <code>val</code>, so it could be elided? Or are you suggesting that <code>r</code> may need used across the <code>.await</code> barrier? I'm not actually sure how to think of reference lifetimes in self-referential structs, because the language tries quite hard to disallow their creation.</p>
<p>As for <code>Part A</code> - I will note that any of the 3 specified options should individually allow us to solve the class of problem I proposed, but I think option C has the most potential utility.</p>



<a name="245661527"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/Async%20Derivation%20and%20Serializable%20Futures/near/245661527" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Kestrer <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/Async.20Derivation.20and.20Serializable.20Futures.html#245661527">(Jul 12 2021 at 09:07)</a>:</h4>
<blockquote>
<p>only the outer layer gets exposed to the proc-macro, and leaves stubs for the child futures?</p>
</blockquote>
<p>But how would the child futures implement the traits? They would need to be derived somehow, but derivation on compiler-generated self-referential types is tricky.</p>
<blockquote>
<p>Additionally, isn't <code>r</code> just an alias of <code>val</code>, so it could be elided?</p>
</blockquote>
<p>In this case that's right, <code>r</code> can (and I think will, because of NLL) be elided, but in the general case that's not possible.</p>
<p>The current future being <code>await</code>ed isn't enough to losslessly serialize and deserialize an <code>async</code> block's state. Take for example this:</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="k">async</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">5</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="n">tokio</span>::<span class="n">task</span>::<span class="n">yield_now</span><span class="p">().</span><span class="k">await</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="n">x</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>In order to recover the future entirely, you would have to serialize both <code>x</code> and the <code>impl Future&lt;Output = ()&gt;</code> returned by <code>yield_now()</code>.</p>
<p>Can you clarify what you mean in option C? To be honest I don't understand it <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span>.</p>



<a name="245670743"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187312-wg-async-foundations/topic/Async%20Derivation%20and%20Serializable%20Futures/near/245670743" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Zoey <a href="https://rust-lang.github.io/zulip_archive/stream/187312-wg-async-foundations/topic/Async.20Derivation.20and.20Serializable.20Futures.html#245670743">(Jul 12 2021 at 10:55)</a>:</h4>
<p>Assuming it isn't elided, I'd expect the state machine to look something like:</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="k">enum</span> <span class="nc">State</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">  </span><span class="n">Start</span><span class="p">,</span><span class="w"></span>
<span class="w">  </span><span class="n">Yield</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="w"> </span><span class="n">YieldFuture</span><span class="p">),</span><span class="w"></span>
<span class="w">  </span><span class="nb">Result</span><span class="p">(</span><span class="n">X</span><span class="p">),</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>Threading state across <code>.await</code> frames  through the state machine in the same way you would allocate registers for a computation using <a href="https://www.haskell.org/arrows/biblio.html#Hug00">John Hughes' Arrow notation</a>.</p>
<p>As for option C, it's something approximating automated bounds discovery by finding places where the addition of a trait which could be derived would satisfy typecheck on <code>async fn</code> <code>impl</code>/instantiation sites, and attempting to derive them for that generator.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>